Multimedia messaging method and system using watermarks

ABSTRACT

A system for and method of controlling retransmission of a content item contained in a multimedia message. The method comprises: receiving the message containing the content item from a sender together with an identifier of an intended recipient of the message, processing the content item to detect the presence or absence of a watermark therein, if the absence of a watermark has been detected, *causing a watermark to be embedded in the content item, and* allowing retransmission of the message including the watermarked content item to the intended recipient, and otherwise controlling retransmission of the message including the content item to the intended recipient. Controlling may comprise e.g. charging a fee for the retransmission, restricting forwarding of the watermarked content item or disallowing the retransmission. The invention relates to the Multimedia Messaging Service (MMS). Basically, the invention proposes “tagging” the content transmitted in a multimedia message with a watermark before sending it to the recipient. Preferably, such tagging is done in the store-and-forward component of the system. In the system as described above this would be the MMS  120.

The invention relates to a method of controlling retransmission of acontent item contained in a multimedia message. The invention furtherrelates to a system arranged for controlling retransmission of a contentitem contained in a multimedia message.

Multimedia Messaging Service (MMS) is a standard that lets users sendand receive multimedia messages with formatted text, graphics, audio andvideo clips. MMS messages can be sent to (MMS enabled) mobile phones orstandard e-mail addresses. The sending entity can for instance be amobile phone or an application running on a computer.

FIG. 1 shows the generic MMS Person-to-Person flow. A first person wantsto send an MMS message to a second person. He takes his mobile phone 101or other MMS user agent, composes the message and includes one or moremultimedia objects in the message. After hitting the “send” button, themultimedia message (text and the multimedia objects) is transmitted overa mobile network 110 to a Multimedia Messaging Service Center (MMSC)120.

The mobile network 110 could be e.g. GSM, GPRS, UMTS or a comparablesystem. As multimedia messages can potentially grow very big, abroadband mobile network is preferred. The first person could also havecomposed the message on his personal computer connected to the Internet.The message would then have been sent to the MMSC 120 over the Internet.Other ways to transmit multimedia messages are of course also possible.It is important to note that the message is always sent to the MMSC 120and not directly to the recipient. This architecture is an example of astore-and-forward messaging system.

The multimedia message is received by the MMSC 120. From there it isforwarded to the intended recipient (the second person). The multimediamessage is once more transmitted over a mobile network 130, this time tothe mobile telephone or other user agent 141 of the second person. Uponarrival of the multimedia message, this mobile telephone processes thecontent and presents the message to the second person. The two mobilenetworks 110, 130 in question may or may not be the same.

In the above example, the first person used the second person'stelephone number as the address to which the multimedia message was tobe sent. He could also have used the second person's e-mail address. Themultimedia message would then have been forwarded by the MMSC 120 to thesecond person's e-mail box, from which the second person could have readit using a standard e-mail client.

FIG. 2 shows the generic MMS Application-to-Person flow. Multimediamessaging is of course not restricted to messages sent by persons topersons. Many automated services can be offered. For instance, a popularservice is the making available of ringtones or logos for mobiletelephones. In such an application, a user can send a request for aparticular ringtone or logo to a server from his mobile telephone. Anapplication 201 on the server retrieves the appropriate ringtone or logoand transmits it as a multimedia message to the MMSC 120. Once themessage arrives at the MMSC 120, it is handled and forwarded in the sameway as described above with reference to FIG. 1. When the mobile phone141 receives the multimedia message, it extracts the ringtone or logo.

Delivering ringtones and logos is already very popular with the standardtext-based Short Messaging Service (SMS). However, MMS has a muchbroader application. A potential use of Application-to-Person MMS is theprovision of (high) value content. An example could be a service whichcan provide the latest music samples or video clips. The user pays anextra amount for that message to cover the content licensing costs, asmost of this content is copyrighted or protected by other intellectualproperty rights.

A problem occurs if both presented models are combined: The user whopaid for the content (application-to-person) can forward this message(and thus the content) to another user (person-to-person), only payingfor the transmission of the message, but not paying any license fees duefor forwarding the content.

It is an object of the invention to be able to restrict or otherwisecontrol the forwarding of multimedia content.

This object is achieved according to the invention in a method asclaimed in claim 1, and in a system as claimed in claim 5. Basically,the invention proposes “tagging” the content transmitted in a multimediamessage with a watermark before sending it to the recipient. Preferably,such tagging is done in the store-and-forward component of the system.In the system as described above this would be the MMSC 120.

Restricting forwarding, for example by disallowing retransmission of theentire message, or by allowing retransmission of the message butdisallowing retransmission of the content item, is usually desired.Another option is to allow forwarding of the content item and trackingsuch forwarding. By tracking the content, useful information about thedistribution of the content can be gathered (for marketing purposes),but also business rules can be added to limit this distribution of thecontent. Yet another option is to contact the service provider whooriginally provided the content item and to allow or disallow forwardingbased on the response.

Advantageous embodiments are set out in the dependent claims.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments shown in the drawing, inwhich:

FIG. 1 schematically illustrates a multimedia messaging system in whichmessages from one user are passed via an MMSC to another user;

FIG. 2 schematically illustrates a multimedia messaging system in whichmessages from an application are passed via the MMSC to a user;

FIG. 3 schematically illustrates an preferred embodiment of theinvention in which a watermark embedder is added to the MMSC;

FIG. 4 schematically illustrates an embodiment of the invention in whicha watermark detector is added to the MMSC;

FIG. 5 schematically illustrates a Media Mediation System coupled to theMMSC; and

FIG. 6 schematically illustrates an embodiment of the MMSC.

Throughout the figures, same reference numerals indicate similar orcorresponding features. Some of the features indicated in the drawingsare typically implemented in software, and as such represent softwareentities, such as software modules or objects.

An illustration of a preferred embodiment of the invention is shown inFIG. 3. In this embodiment, the MMSC 120 is extended with a watermarkingsystem 130 through which all content present in multimedia messages mustpass. The watermarking system 130 detects the type of content in themultimedia messages (e.g. audio, video, text and so on), chooses anappropriate watermarking scheme for the detected type of content, andembeds a marker in the content using the chosen watermarking scheme.Preferably this is done only if the source of the multimedia message isthe MMS application 201. This way content generated by the user agent101 and sent to the user agent 141 will not be affected.

The marker does not have to be very big, and does not need to contain alot of information. As will become apparent below, the most importantaspect is that it is possible later to determine that the content hasbeen watermarked. The content of the watermark is not very important,but it must be very difficult or impossible to remove a watermark fromthe content. This makes robust watermarking techniques the best choice,as they typically have a low payload but are very difficult to remove.

Of course it is not forbidden to add meaningful content as the payloadof the watermark. For example, the name of the entity that added thewatermark, or an identifier for the recipient of the multimedia messagecould be embedded. An identifier for the provider of MMS application 201could also be embedded, for example as a randomly chosen number that isthen stored in a database and associated with the identity of theprovider.

The MMSC 120 is also provided with a watermark detector, as shown inFIG. 4. Before adding a watermark, the content of the multimedia messageis passed through the watermark detector 410. If a marker can be found,this indicates that the content has passed through an MMSC previously.This is proof that the content is being forwarded. As such forwarding isto be prevented or at least restricted, the MMSC 120 can now takeappropriate action.

The simplest action is to simply delete the multimedia message ratherthan forwarding it. However, this may not be acceptable to users.Usually one part of the message (such as a picture or ringtone)represents forwarded content. Other parts, such as accompanying textmessages or pictures, are original content. The original content shouldnot be blocked simply because there is also a forwarded part in themessage. The MMSC 120 could strip out this forwarded piece of contentand pass on the remaining parts of the multimedia message. Preferably,any parts that have been stripped out are replaced by a messageindicating that they have been stripped out and why.

Alternatively, a license fee required for forwarding the content couldbe computed. This license fee can then be charged to the sending user'saccount, if this information is available. This could work well e.g.with music content, as music rights clearinghouses are available tonegotiate a general compensation or license fee.

The MMSC 120 could also convert the content to a relatively low qualityversion before forwarding the content to the intended recipient. Thisway, the intended recipient would still get the message and the content,but because of the low quality it would not be very valuable. He wouldthen be encouraged to download the original version from the MMSapplication 201 himself.

The action can also comprise reporting the distribution of this contentto a third party such as the service provider. Distribution can then bemade conditional upon approval from the provider. This embodiment worksbest if an identifier for the provider is embedded in the content item.

Another example is the allowance of the forwarding, but billing thesending user a premium price for it. In mobile networks, the sendinguser can be identified e.g. through the telephone number to allow suchbilling. Price information can be appended to the message or be suppliedin a separate message.

If a meaningful payload is present in the watermark, the watermarkdetector 410 can extract this payload and take appropriate action basedon the extracted information. For example, if the extracted informationidentifies the initial recipient of the content being forwarded (i.e.the person who received it from the MMS application server in FIG. 2),the MMSC 120 could compare this identifier against an identifier for thesender of the intercepted multimedia message. If these two are the same,it could permit forwarding. This way, the initial recipient can forwardcontent to his friends, but those friends cannot pass the message on toanyone else.

Alternatively, the identifier could be used to track down the initialrecipient of the content, so that he can be charged a fine, or so thathe can be blocked from using the MMS application.

If the extracted information identifies the MMSC which added thewatermark, or the operator of one or more MMSCs, the MMSC 120 couldallow forwarding of the message if it were the same MMSC as the MMSCwhich added the watermark. This permits forwarding only to and fromusers of the same MMSC.

The watermarking system 310 and the watermark detector 410 can beembodied as separate devices to be installed in the MMSC 120, or asseparate devices connected to the MMSC 120. For example, the system 310could be designed to operate as a proxy to be connected between MMSapplication 201 and MMSC 120, making operation transparent. Similarly,the detector 410 can be designed as a proxy between MMSC 120 and mobilenetwork 130. The devices 310, 410 can be embodied as one single device.They can be embodied in whole or in part in hardware or in software.

The entity operating the MMSC 120 is able to optimize content at theMMSC 120 for the receiving user agent using media mediation system 510,as shown schematically in FIG. 5. Optimization can be the adaptation ortranscoding of the format (for images e.g., GIF to JPEG), the size (forvideo e.g., from CIF to QCIF), the bitrate (for audio e.g., 128 kbps to20 kbps) and other parameters. Since this media mediation system 510 hasaccess to, and knowledge about the multimedia content of the message,the watermark embedding and detection systems 310, 410 are thenpreferably positioned in this system 510.

An illustrative embodiment of the MMSC 120 is shown schematically inFIG. 6. The functionality of the MMSC 120 can be divided conceptuallyinto a server portion 601 and a relay portion 651. Briefly speaking, theserver portion 601 provides the store-and-forward architecture and therelay portion 651 transcodes and delivers the messages.

A message is received by the receiver component 610 of the server 601.The message is processed to detect how it should be handled and to whomit should be forwarded. In accordance with the invention, the contentsof the message is processed by the watermark detector 310 to detectwhether the multimedia content of the message has already beenforwarded. If such detection is positive, the MMSC 120 takes appropriateaction, as explained above.

If forwarding of the message is acceptable, there are several possiblepaths the message can take. Usually the message will be passed on to therelay 651. The relay 651 comprises the watermark embedder 410 and themedia mediation system 510 which transcodes the message to a formsuitable for handling by the recipient. The message is then passed on tothe mobile network 130 so that it can be delivered to the intendedrecipient.

The message can also be passed on to an SMTP interface 620 connected toa network 621 such as the Internet. This interface formats themultimedia message in accordance with Internet e-mail standards, andthen forwards the message as an Internet e-mail message to the intendedrecipient. The SMTP interface 620 may or may not embed a watermark inthe multimedia message before forwarding it. If the SMTP interface 620watermarks the content, further forwarding of the resulting e-mailmessage to another recipient via the MMS network becomes impossible.However, as the Internet e-mail network is not very strictly regulated,it will be very difficult to enforce forwarding restrictions over thee-mail network itself.

If the recipient is not on the same mobile network as the sender of themultimedia message, the relay cannot forward the message itself to theintended recipient. In this case, the server passes the message on to anMMSC 630 which is connected to the correct mobile network. It is thenthe responsibility of this other MMSC 630 to embed the appropriatewatermark before forwarding the multimedia message to the intendedrecipient.

The watermark detector 410 can also be positioned in the relay portion651 of the MMSC 120. This has the advantage that the detector 410 andembedder 310 could e.g. be realized as a single device. However, if aparticular multimedia message is then passed on to another MMSC whichdoes not have a watermark detector in accordance with the invention,there is a risk that the content is forwarded multiple times, which isnot permitted.

Watermark embedding can also be implemented as part of the transcodermodule 510. Content received by an MMSC from an MMS application willoften be of relatively high quality. As most mobile devices will not beable to directly handle such high quality content (due to e.g. lowresolution displays, low quality loudspeakers or headphones),transcoding to a format suitable for the mobile device is necessary.

By making the watermark embedding part of the transcoding process, onecan make sure that the watermark survives the destructive operationsassociated with the transcoding process. For example, if a 24-bitpicture in the TIFF format is converted to an 8-bit grayscale GIF image,a watermark that is embedded in the color components of the originalimage can be severely distorted. If however, the watermark were embeddedafter the conversion to 8-bit grayscale but before the compression tothe GIF format, the final picture will be correctly watermarked.

The functionality of the transcoding module 510 can be extended to morethan just converting between formats, sizes, bitrates etcetera. Forinstance, an interface with MMS applications could be provided. Thisway, an MMS application could for instance dynamically generate contentspecifically adapted for one particular mobile device.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims. The invention is of course applicableto any store-and-forward system, not just the MMS system describedabove. For example, the watermark embedder 310 and detector 410 could beincorporated in the user agents 101, 141.

It is for instance also possible to implement the watermark detectionand embedding functionality according to the invention in an Internet orIntranet e-mail server system. This way e-mail messages cannot be usedto indefinitely forward multimedia content. Analysis of the contents ofe-mail attachments is already done by anti-virus software, and a similarapproach could be used to implement watermark detection in attachments.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim. The word “comprising” does notexclude the presence of elements or steps other than those listed in aclaim. The word “a” or “an” preceding an element does not exclude thepresence of a plurality of such elements. The invention can beimplemented by means of hardware comprising several distinct elements,and by means of a suitably programmed computer.

In the device claim enumerating several means, several of these meanscan be embodied by one and the same item of hardware. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage.

1. A method of controlling retransmission of a content item contained ina multimedia message, the message originating with a sender whichreceived the content item from a provider, comprising: receiving themessage containing the content item from the sender together with anidentifier of an intended recipient of the message, processing thecontent item to detect the presence or absence of a watermark therein,if the absence of a watermark has been detected, causing a watermark tobe embedded in the content item, and allowing retransmission of themessage including the watermarked content item to the intendedrecipient, and otherwise controlling retransmission of the messageincluding the content item to the intended recipient.
 2. The method asclaimed in claim 1, in which controlling retransmission comprisesallowing retransmission of the message including the watermarked contentitem to the intended recipient and billing the sender a premium price.3. The method as claimed in claim 1, in which controlling retransmissioncomprises allowing retransmission of the message including thewatermarked content item to the intended recipient and recording detailsregarding the retransmission.
 4. The method as claimed in claim 1, inwhich controlling retransmission comprises restricting retransmission ofthe message.
 5. The method as claimed in claim 4, in which restrictingretransmission comprises disallowing retransmission of the message. 6.The method as claimed in claim 4, in which restricting retransmissioncomprises allowing retransmission of the message but disallowingretransmission of the content item.
 7. The method as claimed in claim 4,in which retransmission is conditional upon approval by a provider ofthe content item.
 8. A system arranged for controlling retransmission ofa content item contained in a multimedia message, the messageoriginating with a sender which received the content item from aprovider, comprising: receiving means for receiving the messagecontaining the content item from the sender together with an identifierof an intended recipient of the message, watermark detecting means forprocessing the content item to detect the presence or absence of awatermark therein, and for signaling to conditional retransmitting meansthe presence or absence of a watermark, the conditional retransmittingmeans being arranged for, conditional upon receiving a signal indicatingthe absence of a watermark, activating watermarking means for embeddinga watermark in the content item, and activating retransmitting means forretransmitting the message including the watermarked content item to theintended recipient, and for otherwise controlling retransmission of themessage including the content item to the intended recipient.
 9. A mediatranscoding system, arranged for transcoding content items in amultimedia message to a format suitable for an intended recipient,comprising the system of claim
 8. 10. A computer program productarranged for causing a processor to execute the method of claim 1.